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Abstract 

Our  goal  is  to  find  relationships  between  social/cognitive  networks  and  communications 
networks  to  devise  strategies  to  maximize  various  perfonnance  metrics  within  these 
networks.  We  consider  the  case  of  a  group  of  users  that  are  performing  a  particular 
decision-making  objective  by  passing  information  through  a  social  network  that  uses  a 
communication  network  with  errors  and  delays  in  the  transmission  of  information.  The 
quality  of  service  of  the  communication  network  will  adversely  affect  overall  user 
performance  and  trust  relationships  within  the  users  by  changing  the  dynamics  of 
information- sharing  within  the  network.  However,  if  the  affected  users  are  able  to  adapt 
to  these  network  errors,  they  may  be  able  to  maintain  the  same  level  of  perfonnance 
when  compared  to  their  performance  in  an  optimally  performing  network.  We  are 
interested  in  characterizing  the  effects  of  these  errors  and  determining  if  these  dynamics 
can  be  compared  to  properties  of  traditional  communication  networks.  We  are  currently 
exploring  the  use  of  ELICIT  to  examine  the  effect  of  communications  errors  and  delays 
on  user  perfonnance.  We  use  existing  capabilities  within  ELICIT  by  altering  the  sense¬ 
making  agent  parameters  to  mimic  communication  enors  (failure  to  share  factoids)  and 
delays  (delay  in  sharing  factoids).  Further,  we  are  interested  in  the  effect  of  connectivity 
within  the  organization  structure  for  the  ELICIT  experiments.  This  is  a  study  using 
ELICIT  from  a  perspective  that  concentrates  on  the  relationship  of  the  communication 
network  with  the  social  network. 
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1.  Introduction1 

Within  social  networks,  the  underlying  communication  network  connecting  the  users  of 
the  network  may  dictate  the  perfonnance  of  a  group  in  executing  certain  tasks.  The 
connectivity  and  quality  of  service  of  these  networks  results  in  a  varying  degree  of 
mission  performance  among  the  users  of  the  network.  Conversely,  dynamics  within 
social  networks  may  affect  the  perfonnance  of  a  group  despite  a  communication  network 
with  near-perfect  quality  of  service.  Issues  with  trust  between  individuals  or  lack  of 
experience  or  teamwork  will  hamper  mission  perfonnance  and  completion.  This  paper 
considers  the  relationship  between  the  social  networks  and  communication  networks  and 
how  these  interactions  affect  individual  and  group  performance. 

We  have  used  the  Experimental  Laboratory  for  Investigating  Collaboration,  Infonnation- 
sharing,  and  Trust  (ELICIT)  to  conduct  command  and  control  (C2)  experiments  of 
humans  and  human-agent  models.  ELICIT  is  a  configurable  software  platform  designed 
to  measure  the  behavior  of  social  networks  in  a  command  and  control  information¬ 
sharing  scenario.  Participants  in  ELICIT  experiments  are  periodically  provided  with 
“factoids”  or  snippets  of  information.  These  factoids  are  sent  and  received  among  the 
participants,  or  the  participants  can  retrieve  information  from  a  set  of  simulated  websites 
or  databases.  This  information  is  used  to  deduce  infonnation  of  a  fictional  terrorist  threat 
(“identify”ing  the  WHO/WHAT/WHERE/WHEN  of  the  threat).  ELICIT  is  designed  to 
study  the  organization  of  social  networks  and  the  interactions  within  these  networks. 

In  this  paper,  we  study  the  performance  of  social  networks  by  using  ELICIT  in  an 
approach  outside  of  the  typical  set  of  research  using  ELICIT.  ELICIT  aims  to 
demonstrate  though  experimentation  the  effectiveness  of  variations  in  organizations,  as 
suggested  by  Alberts  [Alberts  2003],  where  it  is  hypothesized  that  edge  (or  flat 
topologies)  organizations  will  outperfonn  hierarchical  organizations  in  command  and 
control  scenarios  with  uncertainty,  complexity  and  requirements  for  agility.  ELICIT 
allows  for  human-in-the-loop  or  agent-based  experimentation  with  various  organizations 
(including  edge  and  hierarchical  organizations).  The  underlying  communications 
infrastructure  is  assumed,  but  not  explicitly  investigated.  In  this  paper,  we  have 
manipulated  the  existing  capability  of  ELICIT  to  simulate  the  effects  of  communication 
networks  within  these  experiments.  This  paper  investigates  the  effect  of  communication 
networks  on  the  task  completion  objective  within  ELICIT. 


1.1  Connections  with  Network  Science 

In  this  paper,  we  consider  the  effects  of  communication  networks  on  social/cognitive 
networks.  The  motivation  of  this  work  is  rooted  in  emergence  of  the  field  of  network 
science  as  an  area  of  multidisciplinary  research  [NRC  2005].  In  high-level  terms,  the  goal 
of  this  research  is  to  understand  networks,  in  particular,  the  communication,  information 
and  social/cognitive  networks,  which  are  referred  to  by  the  network  science  stack. 


1  The  authors  would  like  to  acknowledge  Mary  Ruddy  (Parity  Communications,  Inc.  and  Meristic,  Inc.) 
for  providing  ELICIT  configuration  files  and  for  her  countless  discussions  to  develop  this  work. 


4/21 


Broadly  speaking,  communication  networks  refer  to  the  electronic  devices,  infrastructure, 
protocols  and  algorithms,  and  the  transmitted  of  signals  throughout  the  network.  Within 
the  field  of  telecommunications,  the  communications  network  refers  to  the  physical,  data 
link  and  network  layers  of  the  open  systems  interconnection  (OSI)  network  stack 
[Zimmermann  1980],  In  terms  of  ELICIT,  the  communication  network  is  assumed  to  be 
the  infrastructure  allowing  factoids  to  be  shared  and  posted/pulled  to  the  website. 
Currently,  ELICIT  employs  a  perfect  and  instant  communications  infrastructure.  The 
social  and  cognitive  network  includes  the  interactions  between  humans  and  the  cognitive 
processing  of  individuals.  In  terms  of  interactions  between  humans,  ELICIT  considers  the 
organization  of  the  group  and  the  ability  of  the  individual  to  determine  the  terrorist  plot 
from  the  (sub)set  of  factoids.  Social/cognitive  networks  include  how  information  is 
presented  and  processed  by  individuals  and  groups.  We  consider  the  information  network 
as  the  network  layer  in  the  network  stack  that  translates  the  information  sent  in  the 
communication  network  to  the  knowledge  processed  by  the  social/cognitive  network.  It 
spans  information  transmission  techniques  (i.e.  error-correction  coding,  compression 
techniques)  and  also  includes  information  sharing  strategies  within  the  organization.  The 
information  network  within  ELICIT  includes  the  way  in  which  the  factoids  are 
distributed,  and  the  ability  of  users  to  share  factoids  directly  with  other  users  or  post  them 
to  websites. 

One  aspect  of  network  science  is  to  better  understand  the  cross-network  relationships.  For 
example,  how  are  social  networks  affected  by  changes  in  communication  networks? 
Additionally,  is  there  a  measurable  set  of  parameters  that  can  be  used  to  model  and 
predict  these  effects?  Certainly  there  are  a  set  of  parameters  that  detennine  these  effects, 
but  perhaps  there  is  a  dominating  set  of  such  parameters.  The  idea  of  modeling  the 
performance  of  communication  networks  is  well  studied,  but  analytical  models  within 
social/cognitive  networks  are  less  common.  Even  more  uncommon  is  the  coupled 
modeling  of  the  communication  and  social/cognitive  networks.  The  network  quality  of 
service  can  be  modeled,  but  the  interaction  with  humans  is  multi-dimensional  and  also 
varies  with  each  user.  Within  social/cognitive  networks,  the  challenge  is  dealing  with  the 
complexity  of  human  cognition.  The  reactions,  experience  and  dynamics  of  each  user  are 
unique  in  each  case.  This  paper  is  an  initial  investigation  of  the  effects  of  communication 
networks  on  the  performance  of  social/cognitive  networks. 

We  present  a  result  of  a  series  of  experiments  using  ELICIT  and  several  aspects  of  its 
agent-based  version  (abELICIT).  The  organization  of  this  paper  is  as  follows.  First,  we 
consider  how  communication  networks  can  be  expressed  in  ELICIT.  We  use  several  of 
the  sensemaking  agent  parameters  to  represent  communication  network  parameters. 
Second,  we  examine  the  scalability  of  abELICIT  in  tenns  of  the  size  of  the  organization. 
Using  this  communication  network  model  within  abELICIT,  we  then  consider  the  effect 
of  loss  and  delay  of  transmitted  messages.  We  also  consider  the  variation  of  network 
topologies  or  organizational  structure  by  examining  the  effect  of  communication  radius 
on  the  performance  of  ELICIT.  The  final  sections  are  a  discussion  of  related  work  and 
conclusions. 
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2.  ELICIT  and  Communication  Networks 

The  focus  of  ELICIT  [Ruddy  2007]  is  to  provide  a  command  and  control  software 
platform  to  run  human-based  experiments  to  examine  the  effect  of  organization  on  an 
information  sharing  scenario.  Recently,  sensemaking  agents  were  developed  to  run 
ELICIT  [Ruddy  2009],  which  enabled  trials  of  ELICIT  to  be  run  without  human 
participants.  This  version  of  ELICIT  is  called  agent-based  ELICIT  (abELICIT),  (for  this 
paper,  ELICIT  will  subsume  both  abELICIT  and  ELICIT).  These  sensemaking  agents  are 
governed  by  a  set  of  parameters,  which  can  be  used  characterize  the  way  the  agent 
processes  and  shares  infonnation  with  other  participants  in  the  experiment.  For  example 
Anderson  [Anderson  2009]  considers  the  effect  of  several  parameters  (propensity  to  seek, 
propensity  to  share,  sharing  modality)  on  the  perfonnance  of  ELICIT.  Agents  and 
humans  are  able  to  participate  in  the  experiments  together;  however,  we  only  consider 
experiments  comprised  solely  of  agents.  It  is  assumed  that  the  sensemaking  agents  are 
valid  models  of  humans  in  these  experiments.  A  validation  of  the  ability  of  agents  to 
model  actual  human  behaviors  was  performed  by  Wynn  [Wynn  20 10]  .When  considering 
communication  networks  within  ELICIT,  it  is  hypothesized  that  the  sensemaking  agent 
parameters  can  represent  communication  network  parameters.  We  alter  sensemaking 
agent  parameters  to  simulate  the  effect  of  a  parameter  in  the  communication  network. 

2,1  Mutual  Vocabulary 

Studying  social  networks  and  communication  networks  invariably  introduces  two  sets  of 
vocabulary.  Tenns  within  the  social/cognitive  and  organizational  behavior  field  and  in 
the  communication  networks  space  are  used  to  represent  the  parameters  with  regard  to 
the  network.  Here,  the  relationship  between  the  vocabulary  used  in  ELICIT  and 
communication  networks  is  clarified  in  Table  1.  These  terms  are  be  used  interchangeably 
throughout  this  paper. 


ELICIT 


Team  Member,  User,  Player 
Participant,  Sensemaking 
agent 

Organization 
Edge  Organization 


Factoid 


Identify 


Communication 

Network 

Nodes 
Topology 
Random  graph 

Packet 

-NA- 


Parameter  Description 


Individual  in  the  trial  with 
certain  interconnections. 

Graph  representation  of  nodes 
and  links  between  nodes. 

Flat  network  where  any  node 
can  communicate  with  any 
node. 

Snippet  of  information  being 
shared  between  nodes. 

Measure  of  situational 
awareness. 


Table  1:  List  of  ELICIT  and  Communication  Network  parameters 
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To  measure  the  performance  of  ELICIT,  we  evaluate  the  correctness  of  the  group  in 
being  able  to  “identify”  the  details  of  the  terrorist  threat.  The  correctness  measure 
represents  a  measure  of  situational  awareness  within  this  scenario.  The  ELICIT  terrorist 
threat  scenario  used  is  “The  Violet  group  plans  to  attack  a  financial  institution  in  Psiland 
on  April  5  at  11:00  AM”.  Correctness  is  measured  by  the  accuracy  of  the  WHO  (Violet 
group),  WHAT  (financial  institution),  WHERE  (Psiland),  and  WHEN  (April,  5,  11:00, 
AM)  in  each  of  the  identifies.  WHO,  WHAT,  WHERE  are  scored  with  0  or  1,  and  WHEN 
has  a  score  of  {0,  0.25,  0.5,  0.75  and  1.0},  allowing  for  partial  correctness.  The  overall 
correctness  score,  C,  a  value  between  0  and  1  is: 

C  =  0.25  (  WHO  +  WHERE  +  WHAT  +  WHEN  ). 


2.2  Factoid  Set 

An  unexpected  factor  of  the  perfonnance  of  the  ELICIT  experiments  emerged  while 
analyzing  the  results.  When  conducting  runs  of  network  sizes  with  variable  size,  new 
factoid  sets  are  created  from  the  original  factoid  set.  The  original  factoid  set  [Anderson 
2009]  consists  of  68  unique  factoids,  distributed  with  information  on  the  WHO,  WHAT, 
WHERE,  WHEN.  There  are  also  “noise”  factoids,  information  that  provides  no  useful 
information.  Additionally,  factoids  are  randomly  seeded  into  the  nodes  within  the 
organization  in  three  waves  (occurring  at  0,  5  and  10  min). 

When  the  factoid  sets  were  constructed  to  create  trials  for  more  than  34  nodes,  multiples 
of  the  original  factoids  were  made.  For  example,  the  17-node  factoid  set  assigns  the  68 
factoids  to  a  particular  node  and  wave  number.  The  factoid  set  for  an  organization  that  is 
a  multiple  of  17  is  constructed  by  copying  the  original  factoid  set,  and  duplicating  the  set 
of  factoids  and  node  numbers  (node  numbers  1-17  are  replaced  by  18-34,  etc...)  as 
necessary.  This  introduces  duplicates  of  the  each  factoid  and  also  the  “noise”  factoids. 
Using  this  approach  to  generate  factoid  sets,  we  observe  inconsistencies  in  the  results  of 
average  correctness  when  trying  to  characterize  the  effect  of  network  scalability  and  also 
when  trying  to  determine  the  impact  of  packet  loss  rates.  As  the  size  of  the  network 
grows,  the  time  to  gain  average  correctness  increases  significantly.  The  reason  attributed 
to  this  unexpected  result  is  the  increased  congestion  and  processing  time  as  a  result  of  the 
increased  number  of  noisy  factoids.  Secondly,  as  packet  loss  increases,  the  network  is 
able  to  maintain  a  level  of  average  correctness.  This  is  attributed  to  the  duplication  of  the 
factoids  throughout  the  network. 

We  consider  this  to  be  an  interesting  result,  as  this  demonstrates  the  effect  of  the 
information  network  on  the  perfonnance  of  the  social/cognitive  network  despite  varying 
QoS  of  communication  network.  However,  this  result  does  not  contribute  to  the  direction 
of  this  paper,  but  is  of  future  consideration.  For  our  experiments,  we  implement  a  factoid 
set  that  seeds  a  total  of  68  unique  factoids,  regardless  of  the  network  size.  Additionally, 
each  of  the  factoids  is  seeded  at  time  0.  This  prevents  intentional  duplication  of  factoids 
and  controls  the  amount  of  noise  inserted  into  the  network. 
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2.3  Geometric  Graphs  and  Organization 

With  ad-hoc  wireless  communication  networks,  it  is  inevitable  that  each  node  will  not  be 
able  to  directly  communicate  with  every  other  node  in  the  network.  Communications 
must  be  sent  through  multi-hop  paths  (sent  through  intermediate  nodes).  These  topologies 
can  be  represented  by  a  geometric  graph  G(n,r).  With  a  graph  of  n  nodes,  there  is  a  link 
between  two  nodes  x,  and  Xj  if  the  distance  between  the  two  nodes  is  less  than 
communication  range,  r.  A  graph  is  connected  if  there  is  a  multi-hop  path  between  every 
pair  of  nodes.  Connectivity  within  random  graphs  has  been  shown  to  exhibit  a  threshold 
effect,  where  G(n,r)  is  connected  with  high  probability  for  r  >  rj  and  not  connected  with 
high  probability  for  r  <  AY. 

Figure  1  shows  this  threshold  effect  for  n  =  51 .  A  set  of  100  network  topologies  of 
G(n,r)  have  been  created  and  the  network  connectivity  is  determined.  This  figure 
indicates  that  the  minimum  communication  radius  to  have  a  random  graph  of  n  nodes 
with  high  probability  is  r  ~  0.27.  Figure  2  is  a  plot  of  the  minimum  such  r  required  for  a 
network  of  n  nodes  to  be  connected  with  high  probability. 


Figure  1.  Probability  of  connectivity  vs.  communication  radius  r  for  n  =  51. 
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Figure  2.  Communication  radius  r  required  for  connectivity  versus  network  size  n. 

Subsequently,  Figure  2  is  used  to  formulate  the  organization  files  in  our  ELICIT 
experiments.  The  most  commonly  studied  organization  structures  in  ELICIT  experiments 
are  hierarchical  and  edge  organizations.  Our  experiments  consider  edge  organizations. 
The  organization  is  created  by  randomly  choosing  positions  of  n  nodes  and  creating 
edges  with  nodes  within  communication  radius  r.  For  each  network  size,  we  use 
communication  radii  that  are  sufficient  to  have  connectivity  with  high  probability.  We 
generate  a  set  of  organizations  for  each  network  size  that  represent  unique  representations 
of  random  graphs  G(n,r).  These  organizations  represent  reasonable  communication 
structure  in  distributed  networking  environments  supporting  command  and  control 
scenarios. 

Additionally,  the  baseline  ELICIT  experiment  allows  for  users  to  post  and  pull  factoids  to 
a  set  of  simulated  websites.  These  websites  serve  as  centralized  locations  where  multiple 
users  can  access  factoids  (broadcast)  as  opposed  to  just  sharing  through  peer-to-peer 
communications.  In  our  ELICIT  experiments,  we  chose  not  to  use  the  websites.  All 
sharing  of  factoids  is  peer-to-peer. 

2.4  Network  Quality  of  Service 

In  addition  to  the  organization,  we  are  interested  in  simulating  communications  with 
variable  network  quality  of  service  (Qos).  Packet  completion  ratio  and  packet  latency 
within  networks  are  two  parameters  by  which  network  QoS  of  communication  networks 
are  measured.  By  varying  the  network  quality  of  service,  we  are  interested  in  determining 
the  effect  on  the  performance  of  the  social  network  in  ELICIT.  These  network  QoS 
parameters  were  simulated  in  ELICIT  trials  with  two  sensemaking  parameters: 
ShareWithFactor  and  SharingPostingMessageDelay .  Table  2  is  a  list  of  the 
parameters  that  are  altered  within  ELICIT  for  our  experiments. 
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Communication  Network 

Sensemaking  Agent 

Probability  of  a  successfully  transmitted 
packet 

ShareWithFactor 

Packet  latency 

SharingPostingMessageDelay 

Network  topology 

Organization 

Table  2:  List  of  Sensemaking  agent  parameters  and  Communication  Network  parameters 


Packet  transmission  ratio  represents  the  probability  that  a  message  that  is  transmitted  is 
successfully  received  by  the  intended  recipient.  This  is  represented  within  ELICIT  by  the 
ShareWithFactor  parameter.  The  interpretation  of  this  parameter  within  the  agent 
model  is  that  for  every  share  an  agent  wants  to  make,  it  actually  shares  it  with  probability 
ShareWithFactor.  This  is  a  parallel  to  the  communication  network  with  the  packet 
transmission  ratio. 

Packet  latency  is  the  time  it  takes  a  packet  to  travel  from  its  source  to  intended 
destination.  The  time  may  include  transmission  and  processing  delays  through  multiple 
hops.  In  ELICIT  the  sensemaking  agent  SharingPostingMessageDelay  is  used  to 
simulate  packet  latency.  For  each  potential  share  or  post,  the  agent  simply  delays  the 
share  or  post  by  SharingPostingMessageDelay  milliseconds. 

The  remaining  sensemaking  agent  parameters  are  held  constant  throughout  the  duration 
of  these  experiments.  However,  several  parameters  are  changed  with  respect  to  the 
original  sensemaking  agent  setup.  To  minim  i/e  processing  delays  and  to  let  the 
communication  delays  dictate  the  performance  of  these  networks,  several  of  the  network 
parameters  are  set  to  value  =  100  (Appendix  A  is  the  baseline  sensemaking  agent 
configuration  file  with  its  associated  parameters.): 

screeningSelectedMessageDelay 
inf ormat ion Process ingDe lay 
socialProcessingDelay 
sharingPostingMessageDelayk 
awareness Process ingDe lay 
determiningKnowledgeNeedsDelay 


3.  ELICIT  Experiments 

In  this  section,  we  investigate  the  influence  of  communication  networks  on  ELICIT 
described  in  Section  2  with  several  experiments  in  ELICIT  using  the  sensemaking  agents. 
First,  we  consider  the  scalability  of  the  task  completion  process  for  a  variable  set  of 
network  sizes.  Then,  we  investigate  the  effects  of  packet  loss  and  delay  within  ELICIT. 
Last,  we  study  the  effect  of  connectivity  with  the  organization  and  encounter  an 
unexpected  behavior. 
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3.1  Scalability 

The  original  ELICIT  experiments  consisted  of  a  group  of  17  individuals  participating  in 
trials.  Since  then,  trials  involving  34  individuals  have  been  made  possible.  The  limiting 
factor  for  human-in-the-loop  experiments  of  larger  sizes  is  acquiring  adequate  testing 
populations.  The  development  of  agents  in  ELICIT  allows  for  trials  of  larger 
organizations  (or  a  mixture  of  agents  and  humans).  The  use  of  larger  networks  of  agents 
may  enable  more  reasonable  scenarios  involving  decision-making  and  command  and 
control.  Larger  networks  may  also  be  able  to  simulate  infonnation  dissemination  among 
larger  non-homogeneous  populations.  To  assess  the  scalability  of  ELICIT,  we  are 
interested  in  detennining  the  maximum  size  agent  organization  that  the  hardware  and 
software  can  handle  and  the  maximum  organization  that  produces  maximum  correctness 
(C  =  1,  i.e.  full  situational  awareness)  within  a  specified  amount  of  time.  The  current 
instantiation  of  ELICIT  runs  the  agents  on  the  same  system  running  the  ELICIT 
administrative  processes.  Therefore,  if  too  many  agents  are  used,  it  is  possible  that  the 
agents  or  the  administrative  processes  may  not  receive  adequate  resources. 

Lor  our  experiments,  we  run  ELICIT  on  a  Dell  Server  1950  (Quad-Duo  Core,  32  GB 
RAM,  200  GB  HDD).  In  terms  of  available  processing  capability,  RAM,  and  I/O,  none  of 
these  resources  were  a  limitation  in  the  experiments.  We  consider  networks  of  n  =  {17, 
34,  68,  1 00,  1 50}  nodes.  In  terms  of  the  generated  log  files,  agent  logging  is  turned  off, 
which  may  reduce  the  strain  on  the  I/O.  Agent  auditing  is  turned  off  by  including  the  line 
“agentauditing  |  false”  at  the  end  of  an  agent  batch  file.  These  agent-based  ELICIT 
trials  are  run  for  variable  amounts  of  time  (from  one  hour  to  16  hours).  The  resources  of 
the  system  are  not  affected  by  the  duration  of  the  trials. 

The  experiments  are  run  with  organizations  of  the  stated  network  sizes  to  study  the 
scalability  of  ELICIT.  In  tenns  of  the  configuration  of  the  trials,  the  connectivity  of  the 
agents  was  created  using  communication  radii  according  to  Ligure  2,  which  detennines 
the  organization.  Each  subsequent  trial  uses  another  random  instance  of  G(n,r).  The  same 
sensemaking  agent  is  used  for  each  node  in  the  network.  We  employ  sensemaking  agent 
parameters  to  create  a  packet  latency  of  8  seconds,  and  no  packet  loss.  The  trials  were  run 
for  10  hours  and  the  average  correctness  was  determined  after  one  and  two  hours.  Ligure 
3  shows  the  diminished  average  correctness  versus  time  as  n  increases.  Note  that  the 
discontinuity  of  correctness  at  t  =  15  min  is  due  to  the  system  allowing  an  identify  to 
occur  only  after  15  minutes  have  elapsed  in  the  trial.  Also,  Ligure  4  shows  a  plot  of 
correctness  after  one  C(1)  and  two  hours  C(2)  as  a  function  of  n.  A  threshold  effect  is 
present  in  C(1),  C(2)  as  a  function  of  n  around  80  nodes.  We  also  consider  the  time 
required  to  achieve  full  correctness  in  the  network.  In  Ligure  5,  we  plot  the  time  required 
to  achieve  average  correctness,  t  when  C(t)  =  0.99.  Additionally,  we  show  the  time 
required  for  the  node  with  the  maximum  correctness  value  to  achieve  full  correctness. 
One  standard  deviation  is  also  shown  for  each  data  point.  These  two  plots  are  show  as  a 
function  of  n.  One  standard  deviation  is  shown  in  the  error  bars  of  this  figure. 
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Given  the  feasibility  of  running  scenarios  with  large  network  sizes,  the  challenge  is  to 
determine  if  the  network  performance  can  attain  a  certain  level  of  performance  within  a 
given  time  period.  The  implication  of  scalability  in  command  and  control  scenarios  is  that 
decision-making  tasks  must  be  completed  within  a  certain  time  period.  Decision-making 
capability  and  situational  awareness  must  be  attained  to  adequate  levels  of  performance 
in  command  and  control  scenarios.  We  have  also  shown  the  performance  of  the  highest 
achieving  node  and  its  degradation  as  the  network  size  increases.  In  this  scenario,  the 
time  required  increases  by  30  minutes,  whereas  the  time  required  for  the  average 
performance  of  the  network  triples.  In  some  situations,  this  is  the  metric  that  is  of  interest. 
However,  in  these  experiments,  the  node  with  the  maximum  performance  is  not  specified 
and  also  may  change. 


Figure  3.  Correctness  vs.  time  (min)  for  n  =  {  17,  34,  68,  100,  150  }. 
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Figure  4.  Correctness  after  one  and  two  hours  vs.  network  size. 


Figure  5.  Time  required  for  C  =  0.99  vs.  n  for  network  average  and  for  maximum  node. 


3.2  Packet  Loss  and  Delay 

We  now  present  results  on  the  ELICIT  simulations  regarding  packet  loss  and  delay. 
These  results  are  capable  of  being  used  in  the  design  of  communication  networks  that  are 
supporting  social  networks  in  C2  environments.  The  network  design  process  can  take 
these  parameters  into  account  to  optimize  the  mission  completion  measure  in  situations 
where  loss  and  delay  are  controllable  in  the  network.  For  these  two  network  QoS 
parameters,  a  high  packet  completion  ratio  will  likely  result  in  higher  packet  latency,  and 
a  lower  packet  completion  ratio  will  yield  lower  packet  latency.  In  this  way,  the  optimal 
achievable  combination  of  the  two  can  be  designed  to  yield  the  maximum  social  network 
performance.  This  is  also  an  example  of  the  cross-network  analysis  of  communication 
networks  and  social/cognitive  networks. 

Our  set  of  experiments  consists  of  implementing  packet  delays  and  packet  loss  ratios.  The 
trials  consist  of  organizations  of  68  nodes,  and  the  organization  is  constructed  using 
communication  radius  according  to  Table  2.  These  trials  are  run  for  2  hours,  and  the 
correctness  after  one  C(1)  and  two  hours  C(1)  was  measured.  We  have  shown  that  the 
baseline  (8  second  packet  latency,  0%  packet  loss)  achieves  full  average  correctness  after 
approximately  2  hours.  By  incorporating  packet  delay  and  loss  in  the  communications, 
the  average  correctness  is  degraded.  This  set  of  experiments,  listed  in  Table  3,  illustrates 
the  impact  that  packet  delay  and  packet  loss  has  on  average  correctness.  First,  we  show 
the  effect  of  packet  latency  on  the  correctness  of  the  ELICIT  decision-making  task.  Then, 
we  show  the  effect  of  packet  loss  on  the  performance  of  ELICIT.  Last,  we  show  the 
combined  effect  of  loss  and  delay  on  the  performance.  This  set  of  experiments  provides 
insight  into  which  parameter  has  more  influence  on  the  correctness  measure. 
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Network  Parameter 

Values 

Packet  latency 

8s,  15s,  30s,  lmin.,  2  min.,  and  5  min. 

Packet  loss 

{0,  20,  40,  60,  80,  90}% 

Table  3:  Packet  loss  and  latency  values  used  in  ELICIT  experiments 


To  illustrate  the  effect  of  packet  delay  on  correctness  within  ELICIT  trials,  we  vary  the 
packet  latency  from  8  seconds  to  5  minutes.  In  each  trial,  one  packet  delay  is  used  for 
every  share  transmitted  throughout  the  network.  Figure  6  shows  the  average  correctness 
versus  time,  for  the  set  of  packet  latencies.  In  these  trials,  there  is  no  packet  loss.  This 
figure  shows  the  impact  of  packet  delay  on  the  average  correctness.  Figure  7  shows  the 
average  C(1)  and  C(2)  vs.  packet  delay.  This  set  of  experiments  demonstrates  that  there 
is  a  threshold  effect  for  this  particular  scenario  at  around  a  packet  latency  of  30  seconds 
as  seen  in  Figures  6  and  7. 
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Figure  6.  Correctness  vs.  Time  for  packet  latency  =  8s,  packet  loss  =  0%. 
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Figure  7.  Correctness  vs.  delay,  packet  loss  =  0%. 

To  illustrate  the  effect  of  packet  loss  on  correctness  within  ELICIT  trials,  we  consider 
packet  loss  rates  of  0%,  20%,  40%,  60%,  80%,  and  90%.  In  each  trial,  every  share  is 
successfully  transmitted  with  probability  according  to  the  packet  loss  rate.  Figure  8  is  a 
plot  of  correctness  vs.  time  for  each  of  the  trials  for  the  set  of  packet  losses.  This  shows 
the  degradation  in  performance  as  packet  loss  increases.  Figure  9  shows  average  C(1) 
and  C(2)  versus  time  for  a  set  of  trials,  where  each  plot  line  represents  a  run  for  a 
particular  packet  loss.  In  these  trials,  the  packet  delay  is  8  seconds.  This  figure  shows  the 
impact  of  packet  delay  on  the  average  correctness.  There  is  a  threshold  effect  for  this 
particular  scenario  at  around  a  packet  loss  rate  of  50%  as  seen  in  Figures  8  and  9. 


Figure  8.  Correctness  vs.  time,  packet  delay  =  8  s. 
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Figure  9.  Correctness  vs.  loss  for  Packet  latency  =  8s. 


With  these  results,  both  packet  loss  and  packet  delay  can  be  considered  together.  To 
illustrate  the  effect  of  these  two  network  parameters  on  correctness  within  ELICIT  trials, 
we  consider  each  pair  of  packet  loss  rates  (0%,  20%,  40%,  60%,  80%,  and  90%)  and 
packet  delays  (8  s.,  15  s.,  30  s.,  1  min.,  2  min.,  and  5  min.).  The  same  approach  to  the 
organization  and  use  of  factoids  as  the  last  set  of  trials  was  implemented.  Figure  10 
shows  C(2)  as  a  function  of  both  packet  loss  and  packet  delay.  ELICIT  demonstrates  a 
tolerance  of  packet  loss  in  low  packet  latency  situations.  Once  the  packet  latency  is 
greater  than  around  30  seconds  (as  shown  in  Figure  7),  the  performance  of  ELICIT 
drastically  decreases,  regardless  of  the  packet  loss  rate. 


Figure  10.  Surface  plot  of  Correctness  after  2  hours  for  packet  delay  and  loss. 
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This  is  an  unexpected  result,  as  one  would  believe  that  an  organization  would  be  able  to 
tolerate  packet  delay  as  opposed  to  losses  in  the  transmission  of  packets.  In  the  ELICIT 
experimental  setup,  neither  the  transmitter  (sender)  nor  the  receiver  are  aware  of  a  failed 
transmission.  The  transmitter  is  aware  that  it  did  not  send  the  factoid,  per  the 
shareWithFactor  parameter.  However,  its  behavior  is  not  affected  by  the  knowledge  of 
the  failed  packet  transmission  ( i.e .  the  node  will  not  attempt  to  resend  failed 
transmissions).  Agent  behavior  does  dictate  ELICIT  performance.  An  explanation  for  the 
indifference  of  the  correctness  with  respect  to  the  packet  loss  rate  is  that  the  agent  model 
indicates  that  the  nodes  will  continue  to  attempt  to  share  factoids  (via  the 
propensityToSeek  parameter).  With  respect  to  packet  latency,  the  performance  of  the 
trial  is  affected  for  each  shared  factoid.  The  nodes  in  the  trials  are  unable  to  compensate 
for  the  effect  of  the  latency,  but  due  to  the  redundancy  of  the  sharing  of  factoids,  the 
agents  are  able  to  eventually  recover  from  failed  factoid  shares. 

3.3  Connectivity 

In  this  section,  we  show  the  effect  that  connectivity  of  the  organization  has  on  the 
correctness  measure  within  ELICIT.  To  vary  the  connectivity  within  the  organization,  the 
communication  radius  r  is  varied  when  creating  G(n,r).  In  terms  of  the  communication 
network,  the  cost  of  increased  connectivity  is  increased  energy  consumption.  Figure  1 1 
shows  the  performance  of  the  organization  versus  the  communication  radius  of  the 
organization. 


Figure  11.  Correctness  vs.  Communication  Radius 


The  result  of  this  set  of  ELICIT  experiments  contradicts  the  expected  outcome.  We  found 
that  with  increased  connectivity,  the  average  performance  of  the  organization  decreases. 
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One  would  expect  that  with  greater  connectivity  in  the  network  the  performance  of  the 
network  would  improve.  By  allowing  for  the  nodes  to  share  factoids  with  nodes  farther 
away,  this  requires  less  multi-hop  communications  so  factoids  will  be  disseminated  more 
rapidly. 

The  immediate  explanation  is  that  the  agents  are  suffering  from  “information  overload”. 
Due  to  the  increasing  number  of  neighboring  agents  in  the  organization,  the  nodes  are 
receiving  more  factoids  than  the  agent  is  able  to  process.  This  is  shown  in  Figure  12.  The 
total  shares  received  in  each  5  minute  interval  over  the  two  hour  trial  length  are  plotted 
for  each  of  the  communication  radii  used  to  create  the  topology  of  the  organization.  In  the 
regions  where  r  >  1.0,  the  average  total  number  of  shares  received  is  saturated,  where  the 
nodes  in  the  network  are  receiving  an  average  of  4000  shares  in  the  course  of  the  two 
hour  trial.  When  examining  the  performance  of  the  network  in  Figure  1 1  and  considering 
the  share  behavior  in  Figure  12,  this  indicates  that  the  nodes  are  receiving  and  average  of 
58  factoids  in  a  5  minute  interval,  but  their  performance  does  not  correspond  to  the 
number  of  received  factoids.  This  indicates  that  the  nodes  are  flooded  with  factoids  and 
cannot  process  all  of  the  factoids.  The  phenomenon  of  overwhelmed  agents  explains  the 
observed  behavior. 


Figure  12.  Total  shares  received  in  each  5-minute  interval  for  r=  {0.3,  0.5,  0.7,  0.9,  1.1,  1.3}. 


4.  Discussion  of  Related  Work 

Existing  social  network  studies  use  social  network  analysis  concepts  such  as  the 
centrality,  betweenness  and  closeness  of  networks.  In  these  cases,  the  communications 
structure  is  abstracted.  Houghton  [Houghton  2006]  considers  the  effect  of  structure  of 
organizations  in  various  emergency  operations  from  a  social  networking  analysis  point  of 
view.  Dekker  [Dekker  2002]  uses  an  experimental  platfonn  called  SCUDHunt  to  study 
the  effect  of  organizations,  information  quality  and  coordination  in  networks.  Based  on 
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the  tempo  of  the  information  flows,  quality  of  sensors,  they  determine  optimal 
organization  and  coordination  for  specific  scenarios.  Delays  in  communication  between 
nodes  within  the  command  and  control  network,  but  the  delays  considered  here  are  the 
result  of  information  understanding.  Brehmer  [Brehmer  2009]  considers  self- 
synchronization  within  C2  environments.  An  experimental  platform  called  D3FIRE  was 
used  to  verify  these  ideas  on  self-synchronization. 

The  existing  studies  with  regard  to  ELICIT  consider  social  networking  concepts  and  test 
them  with  regard  to  different  organizations.  Leweling  [Leweling  2007]  studies  the  effect 
of  mental  models  and  learning  effects  within  edge  and  hierarchical  organizations.  Powley 
[Powley  2008,  2009]  considers  the  effect  of  trust  relationships  on  the  performance  of 
ELICIT  in  edge  and  hierarchical  organizations.  Rosinha  [Rosinha  2009]  studies  the  effect 
of  emergent  leadership  in  various  organizations  within  ELICIT.  In  general,  the  majority 
of  ELICIT  experiments  have  considered  organizational  behavior  and  shared  infonnation 
concepts  without  considering  the  underlying  communication  network. 


5.  Conclusion 

This  work  has  used  ELICIT  to  consider  the  effects  of  the  communication  network  on  the 
social/cognitive  network.  We  have  performed  this  work  in  ELICIT  using  a  significantly 
different  approach  than  past  studies.  The  set  of  ELICIT  experiments  that  have  been 
presented  show  the  effect  of  packet  loss  and  packet  latency  on  the  correctness  measure  of 
a  set  of  sensemaking  agents.  It  is  shown  that  packet  latency  has  a  more  significant  effect 
on  the  perfonnance  of  the  network  than  packet  loss.  Further,  we  have  considered 
scalability  of  the  network  size  and  also  the  effect  of  the  connectivity  of  the  organization 
on  the  ELICIT  trials.  We  have  performed  agent-based  ELICIT  trials  with  organizations 
larger  than  done  before.  The  contributions  of  this  work  are  an  investigation  into  network 
science  research  in  using  an  existing  social  networking  experimental  platform  and 
implementing  quality  of  service  aspects  of  a  communication  network.  Further,  we  have 
contributed  to  the  ELICIT  research  community  a  novel  set  of  agent-based  experiments 
that  consider  the  effect  of  several  sensemaking  agent  parameters. 
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Appendix  A:  Sensemaking  Agent  file 

The  sensemaking  agent  is  controlled  by  a  set  of  48  parameters.  The  parameters 
that  we  considered  in  our  study  are  in  bold.  This  is  the  base  sensemaking  agent 
used  in  our  experiments 

SenseMaking  Agent  57  1 

<begin  agent  configuration  parameters> 

SenseMaking  Agent  l.jar 

net . parityinc . ccrp . web . agent . impl . SenseMaking  Agent  1 
readylntervalDelay | Time  interval  to  click  Ready  button  110000 
messageQueueCapacity | Capacity  of  queue  (-1  means  unlimited)  -1 
messageQueueTimeRemainlnQueue | Time  a  factoid  can  remain  in  queue  (-1 
means  unlimited) |-1 

messageQueueNewerBef oreOlder | If  true  then  newer  messages  are  selected 
before  older | false 

selectMessageFromQueueDelay | Select  message  from  queue  delay | 1000 
shareBeforeProcessing | If  true  then  share  message  before 
Processing | false 

postedTypes | PostedTypes | who, what, where, when 
postFactor | PostFactor | 1 
postOutOf Area | PostOutOf Area | true 
shareWithFactor | ShareWithFactor | 1 
sharedTypes | SharedTypes | who, what, where, when 

shareRelevantAccordingToSiteAccess | ShareRelevantAccordingToSiteAccess | f 
alse 

shareAccordingToSiteAccess | ShareAccordingToSiteAccess | false 
isCompetitiveHoarder | IsCompetitiveHoarder | false 
pullFactor | PullFactor | 1 

timeBef oreFirstldentif y | Time  before  the  agent  does  its  first  identity 
(in  minutes) | 15 

minSolutionAreas | The  minimum  number  of  ID  tables  with  some  data | 1 
hasSeenEnoughToIdentif y | HasSeenEnoughToIdentif y | 5 
isGuesser | IsGuesser | true 

isFrequentGuesser | IsFrequentGuesser | false 
idConf idencelevel | IdConf idencelevel |0.49 

partialldentify | Identify  if  there  are  no  some  answers | true 
propensityToShare | PropensityToShare  possible  values  (low,  moderate, 
high,  very  high) |very  high 

shareModalChoice | ShareModalChoice  possible  values  (both,  post  dominant, 
post  only,  peer  to  peer  dominant,  peer  to  peer  only) Ipost  dominant 
screeningSelectedMessageDelay | Screening  selected  message  (message 
processing)  delay|100 

informationProcessingDelay | Information  Processing  delay|100 
socialProcessingDelay | Social  Processing  delay | 100 

sharingPostingMessageDelay | Sharing/ Posting  each  Message  delay | 100 
awarenessProcessingDelay | Awareness  Processing  delay|100 
determiningKnowledgeNeedsDelay | Determining  Knowledge  Needs  delay | 100 
idAttemptDelay | ID  Attempt  delay  120000 
webRequestDelay | Web  Request  (Pull) 19000 

shareWith | List  of  players  with  whom  agent  may  share  (-1  means  share 
with  all  from  organization  configuration  file) |-1 
shareWithWebSites | List  of  websites  with  whom  agent  must 
share | what, when,  who, where 
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propensityToSeek | PropensityToSeek  possible  values  (low,  moderate,  high, 
very  high) |very  high 

minTimeBetweenPulls | If  the  time  since  the  last  pull  is  not  >= 
minTimeBetweenPulls ,  do  not  Pull  (in  milliseconds,  -1  means  ignoring 
this  parameter) 120000 

minTimeBetweenShares | If  the  time  since  the  last  Share  is  not  >= 
minTimeBetweenShares ,  the  agent  should  wait  before  it  Shares  (in 
milliseconds,  -1  means  ignoring  this  parameter) | 5000 

trustlnlndividuals | Trustlnlndividuals  possible  values  (high,  medium, 
distrust,  no  opinion) | l=no  opinion, 2=no  opinion, 3=no  opinion, 4=no 
opinion, 5=no  opinion, 6=no  opinion, 7=no  opinon, 8=no  opinion, 9=no 
opinion, 10=no  opinon, ll=no  opinion, 12=no  opinion, 13=no  opinon, 14=no 
opinion, 15=no  opinion, 16=no  opinion, 17=no  opinion 

trustlnWebSites | List  of  initial  values  of  Trust  for  web  sites.  Possible 
values  (high,  medium,  distrust,  no 

opinion) | who=medium, where=medium, what=medium, when=medium 

reciprocity | Reciprocity  possible  values  (high,  low,  medium,  na, 

none) | l=none, 2=none, 3=none, 4=none, 5=none, 6=none, 7=none, 8=none,  9=none, 

10=none, ll=none, 12=none, 13=none, 14=none, 15=none, 16=none 

primary | Primary  areas  of  interest.  Possible  values:  who,  what,  where, 

when) | who, what, when, where 

secondary | Secondary  areas  of  interest.  Possible  values:  who,  what, 
where,  when) | 

propensityToShareExternal | If  message  is  not  in  area  of  interest,  then 
agent  shares  it  according  to  sharing  preferences  with  probability  = 
propensityToShareExternal | 1 

awarenessProcessingThreshold | If  cumulative  value  of  the  perceived 
message  value  is  more  or  equal  to  this  variable,  then  start  awareness 
processing . | 2 

pullBetweenSitesDelay | Pull  between  sites  delay | 1000 
postBetweenSitesDelay | Post  between  sites  delay | 500 

provideRelevance | Provide  relevance  for  posted  and  shared  messages | false 
provideTrust | Provide  trust  for  posted  and  shared  messages | false 


